Sampling rare fluctuations of discrete-time Markov chains

Adaptive Importance Sampling on Discrete Markov Chains

In modeling particle transport through a medium, the path of a particle behaves as a transient Markov Chain. We are interested in characteristics of the particle's movement conditional on its starting state which take the form of a \score" accumulated with each transition. Importance sampling is an essential variance reduction technique in this setting, and we provide an adaptive (iteratively u...

Lecture 11: Discrete Time Markov Chains

where {Zn : n ∈ N} is an iid sequence, independent of initial state X0. If Xn ∈ E for all n ∈ N0, then E is called the state space of process X . We consider a countable state space, and if Xn = i ∈ E, then we say that the process X is in state i at time n. For a countable set E, a stochastic process {Xn ∈ E,n ∈ N0} is called a discrete time Markov chain (DTMC) if for all positive integers n ∈ ...

Discrete Time Markov Chains : Ergodicity Theory

Lecture 8: Discrete Time Markov Chains: Ergodicity Theory Announcements: 1. We handed out HW2 solutions and your homeworks in Friday’s recitation. I am handing out a few extras today. Please make sure you get these! 2. Remember that I now have office hours both: Wednesday at 3 p.m. and Thursday at 4 p.m. Please show up and ask questions about the lecture notes, not just the homework! No one cam...

Discrete Time Markov Chains 1 Examples

Example 1.1 (Gambler Ruin Problem). A gambler has $100. He bets $1 each game, and wins with probability 1/2. He stops playing he gets broke or wins $1000. Natural questions include: what’s the probability that he gets broke? On average how many games are played? This problem is a special case of the so-called Gambler Ruin problem, which can be modelled using a Markov chain as follows. We will b...

Lecture 10: Discrete Time Markov Chains